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Smart Start Evaluation Plan 



The challenges of conducting a comprehensive evaluation of a program with the breadth 
of Smart Start are numerous. This new North Carolina initiative has brought together parents, 
educators, service agencies, businesses, churches, and policymakers to focus on the needs of 
young children from birth through age five and their families. Community partnerships are 
planning how best to meet their own needs, and have begun improving and expanding previous 
programs and designing and implementing new programs. Broadly, the Smart Start goals 
address 4 major areas: early childhood education, family support services, health services, and 
service integration/sy stems change. Improvements in these areas are aimed toward ensuring that 
all children are healthy and prepared to succeed when they enter school. 

An evaluation team at the Frank Porter Graham Child Development Center has worked 
since the fall of 1 993 to develop a comprehensive plan to evaluate the state and local goals and 
objectives of Smart Start. The team consists of faculty from several schools or departments at 
the University of North Carolina at Chapel Hill— Psychology, the School of Education, the 
Department of Maternal and Child Health in the School of Public Health, and the School of 
Social Work-in addition to researchers based at the FPG Center. Different members have 
expertise in child care, family services, health care, early childhood special education, social 
systems change, program evaluation, qualitative and quantitative research methods, and 
biostatistics. The responsibility of the evaluation team is to plan and conduct a thorough 
evaluation of Smart Start. 

We began by observing the process of local partnerships coming together as teams and 
discussing their goals and objectives. We attended local meetings and statewide County 
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Collaboration meetings to listen to local and state-level participants discuss their ideas for Smart 
Start. We watched and listened as county leaders, DHR core team members, and the NC 
Partnership for Children set 3 1 goals for Smart Start, acknowledging that different counties 
would more intensely focus on different goals. • More recently, a review of the short-term plans 
submitted by the projects to obtain first year funds and the long-term plans submitted June 1 , 
1994, for their continuing funds allowed another opportunity to determine the objectives and 
outcomes to be evaluated. We believe the program’s objectives are now sufficiently clear for us 
to offer an overall evaluation plan. 

Based on our review of the local plans, most local partnerships have developed projects 
within each of the major areas—early childhood education, family support services, health 
services, and service integration and systems change— although their emphases on different areas 
vary considerably. For example, one partnership may be focusing its major efforts on improving 
early childhood education, with less intensive efforts directed toward improving the health status 
of children. In another partnership the emphases may be reversed. Some are funding or 
beginning several projects on a small scale, while others are supporting few but large-scale 
projects. Thus, the evaluation plan must address each of the major areas to accommodate the 
varying emphases across local partnerships, and it must be sensitive to the fact that measurable 
improvements in particular areas will be different from county to county depending on their 
area(s) of emphasis. 

In addition to variation in the area(s) of emphasis, local partnerships vary in their 
selection of specific strategies designed to improve a particular area. For instance, to improve 
the quality of early childhood education, one partnership may implement new teacher 
compensation strategies while another may attempt to improve quality through facility 



improvement grants. Most counties/regions are using multiple strategies, with some unique to 
their own county/region and others similar to other partnerships’ strategies. The process 
evaluation we conduct will document the variation in and number of strategies, but will have no 
control over which families/children receive which services. We will therefore aim the outcome 
evaluation at the more global level, assessing progress toward the major Smart Start goals, rather 
than at the specific strategy level. 

Evaluator’s Responsibilities 

The specific responsibilities of the FPG-UNC evaluation team, as delineated in the 
authorizing legislation and in our contract with the Department of Human Resources, are as 
follows: 

1 . To conduct a formative and summative evaluation; 

2. To emphasize quality of programs as a central component of the evaluation; 

3. To use reliable statistical methods and objective measures of outcomes; and 

4. To provide a detailed fiscal analysis of the use of State funds. 

The formative evaluation to date has included a review of the first 9 months of team-building and 
program implementation, as described in a report distributed in May, 1994. During the summer 
of 1994, 60 key participants in the 12 projects (5 per project) were interviewed and asked to 
reflect on the first year. Their responses have been analyzed and feedback provided to the 
leaders of the County Collaboration process; the interviews will continue to be analyzed for other 
formative information that might be helpful to the State or future Smart Start counties. Other 
formative studies of the process of implementing Smart Start may be conducted, if needed. 

Our fourth responsibility— a detailed fiscal analysis— is being conducted in collaboration 
with the Division of Child Development, which already had a system in place for monitoring the 
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child care subsidy programs. We have worked with DCD staff to finalize a system of monitoring 
the 244 different activities that comprise Smart Start in the first 1 2 partnerships, and will 
supplement their financial data with information about the numbers of programs, families and 
children served by these programs, as well as programs’ abilities to leverage other funds for their 
efforts. 

The rest of this report will describe the components of the evaluation plan that address the 
summative evaluation (item 1 , above), including the emphases on quality (2) and objective 
outcomes (3). The table on the following two pages shows the different strategies that will be 
employed to gather information needed for the evaluation. The pages following the table 
describe each strategy in more detail. We have organized the presentation by the major goals 
developed for Smart Start by the NC Partnership for Children, the DHR core team, and the local 
partnership team leaders. The following information is provided for each goal: specific 
outcomes (i.e.. What behaviors or conditions are expected to change through Smart Start 
efforts?), measurement strategies/techniques (i.e.. What are we going to use to collect the data?), 
sources of data (i.e.. From whom are we going to collect data?), domains covered by the 
measurement strategy (i.e.. What kinds of information are we going to collect?), times of data 
collection (i.e.. When are we going to gather the information?), numbers of subjects assessed 
(i.e.. From how many people are we going to obtain information?), and subject selection process 
(i.e.. How are we going to choose the people or programs from which we gather information?). 
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Summary of Smart Start Evaluation Plan: 
Goals 1 and 2 



Goal 


Expected Outcome 


Measurement Tool 


Source of Data 


1. Children are 
healthy and prepared 
to succeed in school 


more children will be 
prepared to succeed 
when they enter 
school 


Kindergarten Teacher Checklist 


kindergarten teachers 


FirstSTEP Screening Test 


kindergarten children 


local preschool screening 
measures 


public school records 


fewer children will be 
retained in the early 
school grades 


existing database 


SIMS database from NC DPI 


fewer kindergartners 
will be identified for 
the first time as 
having special needs 


existing database 


SIMS database from NC DPI 


higher scores on end- 
of- year test in 3rd 
grade 


existing database 


SIMS database from NC DPI 


more kindergartners 
will be healthy when 
they enter school 


Kindergarten Health Assessment 


public school records 


health status of young 
children (b-5) will 
improve 


existing databases 


state-level health agencies (e.g., 
Medicaid, Health Services 
Information System) 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 


2. Families 
effectively fulfill their 
role as primary 
providers, nurturers, 
and teachers 


families value 
learning by 
participating in 
educational activities 
with their children, 
families have access 
to the support services 
they need and want, 
families act as role 
models through their 
involvement in the 
community, and 
families feel 
empowered 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 
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Summary of Smart Start Evaluation Plan: 
Goals 3 and 4 



Goal 


Expected Outcome 


Measurement Tool 


Source of Data 


3. High quality, 
affordable services for 
children and families 


quality of child care 
will improve for all 
children, including 
those with special 
needs 


Early Childhood Environment 
Rating Scale, Infant/Toddler 
Environment Rating Scale, Family 
Daycare Rating Scale 


classroom observations of selected 
child care centers and family 
daycare homes 


regulatable aspects of 
child care will 
improve 


Child Care Director Interview and 
Family Daycare Home Provider 
Interview; Child Care Worksheets 
from Needs & Resources 
Assessment 


child care center directors and 
family daycare providers 


more child care 
providers will be 
well-trained 


Self-Assessment of Skills and 
Training Needs for Early 
Childhood Professionals 


child care providers in child care 
centers and family daycare homes 


more children with 
special needs will be 
served in inclusive 
settings 


Child Care Director Interview 


child care center directors 


existing database 


Part H and SIMS (Part B) 
databases 


service coordination 
for children with 
special needs will 
improve 


focus groups 


parents of children with 
disabilities and professionals who 
serve children with special needs 


more families of 
children with special 
needs will participate 
in and be satisfied 
with early 

intervention services 


family survey 


families of children with 
disabilities who are receiving early 
intervention services 


community services 
will be more readily 
available, affordable, 
and of high quality 


family interview 


families who use child care; 
families who use community 
services; 

families who participate in Smart 
Start efforts 


4. NC counties value 
children and families 
by providing options, 
resources, and 
encouraging 
collaboration 


inter-agency 
coordination and 
collaboration will 
improve over time 


Network Analysis Survey 


agency administrators and service 
providers 


more services will be 
available, affordable, 
and of higher quality 


family interview 


families who use community 
services and child care 


number of agencies 
that have parents on 
their advisory boards 
will increase 


agency survey; Child Care Center 
Director Interview 


agency administrators; child care 
center directors 
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Goal 1: All North Carolina children are healthy and prepared to succeed in school 

when they enter kindergarten. 



A. Expected Outcome: Each year, a higher proportion of entering kindergartners will be 
prepared to succeed when they enter school. 

1. Measurement Strategy /Tool: Kindergarten Teacher Checklist, a 37-item rating scale 

Data Source: Kindergarten teachers in Smart Start counties will rate samples of 
kindergartners annually. 

Variables/Domains: cognitive, language, socio-emotional and motor development, 
classroom behavior 

Data Collection Times: Teacher Checklists will be distributed mid-semester in the fall, 
once teachers have become familiar with children’s skills. 

Numbers: about 4,000 kindergartners per year in the 12 initial Smart Start projects 

Selection Process: All kindergarten teachers in small counties and randomly selected 
teachers in larger counties will be asked to complete a rating scale on every fifth child on 
their class roll. 

Other Details: The teachers will also complete a question about the proportion of 
children in their class who were adequately prepared for kindergarten. Parent consent is 
not required since individual children will not be identified. 

2. Measurement Strategy /Tool: FirstSTEP Screening Test 

Data Source: individual assessments of kindergartners 

Variables/Domains: cognitive, language, and motor development 

Data Collection Times: in the fall of the year after children are settled into the 
kindergarten routine, approximately November-December 

Numbers: between 1 50-200 children, a subset of those rated by teachers 

Selection Process: In 4 counties all kindergarten teachers at 1-2 randomly selected 
schools will be asked to send home consent forms to the parents of the children who were 
rated in A. 1 . above (every 5th child on the roll). The child of parents who return consent 
forms can be assessed in a 30-minute, individual session with a trained assessor. 
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Other Details: This procedure is a way to test the validity of the teacher checklist. We 
expect that teachers’ ratings and individual children’s scores on the assessment will be 
highly correlated but we need to demonstrate that this is true for the particular teacher 
rating scale we have selected. 



3. Measurement Strategy/Tool: local preschool screening measures (e.g., DIAL-R), 
EPSF, Brigance) 

Data Source: The local education agencies (LEAs) in each Smart Start county that keep 
records of each year’s kindergarten screenings. 

Variables/Domains: “readiness” skills in the areas of cognition, language, motor, socio- 
emotional development 

Data Collection Times: entry into kindergarten 

Numbers: approximately 21,000 kindergartners in the 18 initial Smart Start counties 

Selection Process: We will use aggregate data from all children in the local system who 
received the screening measure. 

Other Details: Since no standard screening approach is used across counties, the 
evaluation cannot obtain baseline and future years of data on readiness from LEAs. 
However, we can look at each county as its own control if the LEA summarizes the 
screening data (not every LEA that screens children summarizes the data). For those 
Smart Start counties in which the LEA(s) summarizes the data, information from 1993 
and 1994 (before Smart Start) can be considered baseline, with data each year thereafter 
collected and summarized by the evaluation team (i.e., mean screening scores or 
percentages of children falling below the 10th or 25th percentile). 



B. Expected Outcome: Retention rates in kindergarten through third grade will decrease over 
time, as more children enter school prepared to succeed. End-of-year test scores in third grade 
should increase over time as more children enter school prepared to succeed. With better early 
intervention services and early identification of children with disabilities, the number of children 
identified for the first time in kindergarten as having special needs will decrease. 

Measurement Strategy/Tool: existing database 

Data Source: SIMS database from NC Department of Public Instruction 

Variables/Domains: retention rates in kindergarten through third grade, initial special 
education placement rate in kindergarten, third grade end-of-year test scores 
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Data Collection Times: end of each academic year 

Numbers: approximately 21,000 children per grade in the 18 initial Smart Start counties 
combined 

Selection Process: Information is available for all children in public school. 

Other Details: Documenting trends in retention rates, special services, and third grade 
achievement will provide global measures of school progress in the years pre- and post- 
Smart Start implementation. 



C. Expected Outcome: Each year, a higher proportion of entering kindergartners will be 
healthy when they enter school. 

Measurement Strategy/Tool: Kindergarten Health Assessment, a 2-page, state- 
mandated health form completed upon school entry 

Data Source: public school files of each entering kindergartner in Smart Start counties 

Variables/Domains: immunization status; presence/absence of uncorrected vision, 
hearing, speech, and dental problems; health or learning disabilities that were undetected 
before school; extreme under- or over- weight; anemia 

Data Collection Times: fall of each year 

Numbers: approximately 21,000 children enter kindergarten each year in the 18 original 
Smart Start counties 

Selection Process: Data will be summarized from all children whose Kindergarten 
Health Assessment are accessible in the school files. 

Other Details: By abstracting data from the Kindergarten Health Assessment each year, 
the evaluation team can monitor trends in the numbers of children who arrive at school 
with undetected health problems. With more and better preschool services, these 
numbers should decline each year. Confidentiality may be an issue, but individual names 
are not needed. 
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D. Expected Outcome: The health status of children under 5— newborns, infants, toddlers, and 
preschoolers— will improve. 

1. Measurement Strategy /Tool: existing databases 

Data Source: birth/death certificate data from Vital Statistics; health status data from 
WIC, Health Services Information System, Medicaid, and DSS 

Variables/Domains: rates of prematurity, LBW, mortality; proportions of children with 
age-appropriate immunizations, obesity or stunting, iron deficiency; rates of EPSDT and 
lead screenings; results of EPSDT screening (sensory, developmental, hematologic); lead 
status 

Data Collection Times: ongoing assembly of the various annually-reported statistics 
Numbers: Numbers will vary depending on the database. 

Selection Process: The birth certificate database covers all children bom in N.C.; the 
WIC and Medicaid databases cover the children enrolled in those programs; the HSIS 
covers health department clients. 

Other Details: Documenting trends over time in the Smart Start counties compared to 
non-Smart Start counties will provide a comparison of outcomes related to health. Within 
the group of Smart Start counties, the different outcomes can be compared to each other 
as predicted by the level of emphasis being placed on particular health efforts. 




2. Measurement Strategy/Tool: Family Interview 
Data Source: families in each Smart Start county/region 

Variables/Domains: children’s health status; utilization of health care; insurance 
coverage; disability/activity limitation; specific health problems; barriers to care 

Data Collection Times: Family interviews will be conducted annually during the fall 
and winter. 

Numbers: Approximately 1,500 families (range in counties, approximately 70-150) will 
be interviewed. 

Selection Process: Families will be selected with 3 different sampling strategies. First, 
the interview, in questionnaire form, will be sent home to parents of children in all child 
care classrooms that will be observed. This sample will enable us to address families’ 
options of child care and the affordability of child care. Second, families in the child 
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service coordination caseloads will be randomly selected. Families in this sample are 
most likely to be receiving multiple community services. This subsample will enable us 
to address issues regarding service availability, satisfaction, and barriers to services. 
Third, each local partnership will have an opportunity to nominate a target group of 
families whom the partnership believes is benefiting directly from one or more of their 
Smart Start efforts. This target group may include families who are participating in a 
particular program or families from particular neighborhoods that have been targeted by 
the partnership. Within the target group, a random sample of families will be 
interviewed. This strategy ensures that the opinions and experiences of families 
benefiting directly from local Smart Start efforts will be represented in the evaluation. 

Other details: All selected families will be asked the same set of questions. In the child 
care sample, the questions will be in the form of a written survey to be completed by 
families. In the other two samples of families, the same set of questions will be asked 
through an interview format (either face-to-face or by phone). Respondents may be asked 
to check immunization records to confirm immunization status. 
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Goal 2: North Carolina families effectively fulfill their roles as the primary providers, 
nurturers, and teachers helping their children reach their full potential. 



Expected Outcomes: Families are more involved in their children’s development, have more 
timely access to the services they need, place a higher value on learning and education for 
themselves and their child, and are more empowered to take control of their own lives. 

Measurement Strategy/Tool: Family Interview 

Data Source: families in each Smart Start county/region 

Domains: family learning activities, participation in parenting activities, service 
availability and use, satisfaction with services, supportiveness of agency service 
providers, barriers to receiving services, choices in child care, child care costs, 
involvement in the community, family strengths, feelings of empowerment 

Data Collection Times: Family interviews will be conducted annually during the fall 
and winter 

Numbers: Approximately 1 500 families will be interviewed. 

Selection Process: Families will be selected with 3 different sampling strategies. First, 
the interview, in questionnaire form, will be sent home to parents of children in all child 
care classrooms that will be observed. This sample will enable us to address families’ 
options of child care and the affordability of child care. Second, families in the child 
service coordination caseloads will be randomly selected. Families in this sample are 
most likely to be receiving multiple community services. This subsample will enable us 
to address issues regarding service availability, satisfaction, and barriers to services. 
Third, each local partnership will have an opportunity to nominate a target group of 
families whom the partnership believes is benefiting directly from one or more of their 
Smart Start efforts. This target group may include families who are participating in a 
particular program or families from particular neighborhoods that have been targeted by 
the partnership. Within the target group, a random sample of families will be 
interviewed. This strategy ensures that the opinions and experiences of families 
benefiting directly from local Smart Start efforts will be represented in the evaluation. 

Other details: All selected families will be asked the same set of questions. In the child 
care sample, the questions will be in the form of a written survey to be completed by 
families. In the other two samples of families, the same set of questions will be asked 
through an interview format (either face-to-face or by phone). 




14 16 



Goal 3: All North Carolina families with children 0-5 have access to high quality, 
affordable services they need and want, including early childhood education, 
services for children with special needs, and other services that support families. 

A. Expected Outcome: The quality of child care centers and family daycare homes will 
improve over the years in Smart Start counties. Specifically, the average total score on a child 
care quality rating scale will increase over time in observed child care centers. 

Measurement Strategy/Tool: Early Childhood Environment Rating Scale (ECERS), 
Infant/Toddler Environment Rating Scale (ITERS), and Family Daycare Rating Scale 
(FDCRS) 

Data Source: Classroom observations of the developmental appropriateness of child 
care centers, using the ECERS and ITERS, and observations of family daycare homes, 
using the FDCRS. 

Domains: personal care routines, furnishings and display, language and reasoning 
activities, fine and gross motor activities, creative activities, social development, and 
adult needs 

Data Collection Times: annually in the fall 

Numbers: 243 child care centers and 120 family daycare will be observed, all within 
Smart Start counties/regions. 

Selection Process: Within the 3 Smart Start counties that have fewer than 20 centers, all 
centers will be asked to participate in the evaluation. In the remaining 9 Smart Start 
counties/region, two samples of child care centers will be selected. The first sub-sample 
will be a random sample of 10 or 20 licensed centers (20 in the larger areas— Cumberland, 
Mecklenburg, and Region A) selected from the Division of Child Development’s 
directory of all licensed centers. The second sub-sample will be a sample of 10 centers 
chosen by each local partnership because of their participation in Smart Start. 

Other Details: Within each center, a preschool classroom (i.e., a class in which the 
majority of children are 3 to 5 years old) will be selected randomly for an ECERS 
observation. If the center also has an infant/toddler room (i.e., a class in which the 
majority of children are between infancy and 2 1/2 years), an ITERS will be completed on 
a randomly selected infant/toddler classroom. 

Each year, a new random sample of child care centers and family daycare homes will be 
selected, testing the hypothesis that child care quality in general is improving. The Smart 
Start selected sample of centers will be asked to participate each year, allowing for 
assessment of quality improvement specifically in centers that have directly participated 
in Smart Start efforts. 
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B. Expected Outcomes: The regulatable aspects of quality (e.g., group size, teacher:child 
ratios) will improve in child care centers and family daycare homes in Smart Start counties. The 
number of child care openings and the days and hours of operation of centers/homes, on average, 
will increase in Smart Start counties, providing families greater accessibility. The education 
level of child care directors and providers in Smart Start counties will improve. 



1. Measurement Strategy /Tool: Child Care Worksheets and Family Child Care 
Provider Worksheets from the Needs and Resources Assessment 

Data Source: phone interviews with center directors and family home providers 

Variables/Domains: teacher/child ratios, group size, capacity, education level, staff 
turnover, A/AA/NAEYC status (or NAFDC for family child care homes), salaries, hours 
of operation 

Data Collection Times: annually in February-March 

Numbers: up to 60 child care centers and family daycare homes in each of the original 
Smart Start counties 

Selection Process: We will repeat the random selection process used initially in the 
Needs and Resources Assessment. 

Other Details: These data were originally collected in the spring of 1994 as part of the 
Needs and Resources Assessment in all 100 counties. They thus serve as baseline 
measures of several indices of child care quality. Annual data collection in Smart Start 
counties will allow for a comparison across time. 



2. Measurement Strategy/Tool: Child Care Center Director Interview and Family 
Daycare Home Provider Interview 

Data Source: interviews with child care center directors and family daycare home 
owners/providers 

Domains: child care costs, days and hours of operation, support services for children and 
families that are available at child care centers, number of children with special needs 
enrolled in child care centers, teacher training and education, wages and benefits, 
director’s education and experience 

Data Collection Times: annually in the fall at the same time ECERS/ITERS/FDCRS 
data are collected (see previous page) 
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Numbers: Child care directors in each of the 243 centers participating in observations 
will be interviewed. Family daycare providers in each of the 120 family daycare homes 
will also be interviewed. 

Selection Process: In centers that participate in the child care quality observation, the 
child care directors will be interviewed. Family daycare home providers who participate 
in the child care quality observations will also be interviewed. 

Other details: Some of the information gathered in these detailed interviews will be the 
same as gathered in the larger random sample of 60 centers/homes per county. However, 
this strategy assures a sample of centers/homes on which the evaluation team has both 
regulatable and process measures of quality, allowing for a study of the interrelationships 
among the two. 



C. Expected Outcome: More child care providers in Smart Start counties will report being 
more knowledgeable of and will have received training in topics related to quality child care, and 
their perceived need for additional training will decrease. 

Measurement Strategy/Tool: The Self-Assessment of Skills and Training Needs for 
Early Childhood Professionals 

Data Source: Child care providers in the classrooms selected for 
ECERS/ITERS/FDCRS observations will complete a self-assessment form. 

Domains: child development, child care environment, curriculum and learning, 
professionalism 

Data Collection Times: annually in the fall at the same time ECERS/ITERS/FDCRS 
data are collected 

Numbers: 243 child care providers from the participating centers and 120 family 
daycare home providers 

Selection Process: Child care providers from the selected classrooms in each of the 
centers that participate in the child care quality observations will complete the self- 
assessment form. All participating family daycare home providers will complete the self- 
assessment form. 
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D. Expected Outcome: Over time, families in Smart Start counties/regions will report that 
more services are available in the community, the services are more affordable, and the services 
are of higher quality. 

Measurement Strategy/Tool: Family Interview 
Data Source: families in each Smart Start county/region 

Domains: service availability and use, satisfaction with services, supportiveness of 
agency service providers, barriers to receiving services, choices in child care, child care 
costs 

Data Collection Times: Family interviews will be conducted annually during the fall 
and winter. 

Numbers: Approximately 1,500 families will be interviewed or surveyed. 

Selection Process: Families will be selected with 3 different sampling strategies. First, 
questionnaires will be sent home to parents of children in all child care classrooms that 
will be observed. This sample will enable us to address families’ options of child care 
and the affordability of child care. Second, families in the child service coordination 
caseloads will be randomly selected. Families in this sample are most likely to be 
receiving multiple community services. This subsample will enable us to address issues 
regarding service availability, satisfaction, and barriers to services. Third, each local 
partnership will have an opportunity to nominate a target group of families whom the 
partnership believes is benefiting directly from Smart Start. This target group may 
include families who are participating in a particular program or families from particular 
neighborhoods that have been targeted by the partnership. Within the target group, a 
random sample of families will be interviewed. This strategy ensures that the opinions 
and experiences of families benefiting directly from local Smart Start efforts will be 
represented in the evaluation. 

Other details: All selected families will be asked the same set of questions. In the child 
care sample, the questions will be in the form of a written survey to be completed by 
families. In the other two samples of families, the same set of questions will be asked 
through an interview format (either face-to-face or by phone). 
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E. Expected Outcome: Access to inclusive placements (i.e., placement of children with special 
needs in regular child care and general early childhood programs in the community) will increase 
as a result of Smart Start efforts. 

Measurement Strategy/Tool: Part H and SIMS (Part B) state agency data bases. 

Data Source: Two existing state agency data bases, the Part H data base from the N.C. 
Center for Environmental and Health Statistics, and the SIMS data base from the NC 
Department of Public Instruction, will be accessed. 

Variables/Domains: diagnosis, eligibility category, severity of disability, primary service 
setting, type of special services being provided, and demographic information 

Data Collection Times: baseline (spring of 1994) and annually during implementation of 
Smart Start 

Numbers: approximately 6,000 Part H eligible children and 9,000 Part B eligible children 

Selection Process: We will compile and analyze data from files on all children ages birth - 
5 who are eligible to receive Part H (Infant- Toddler) or Part B (Preschool) services in North 
Carolina through the two existing state data bases mentioned above. 

Other Details: Part H-eligible children are followed by NC's Developmental Evaluation 
Centers (DECs) throughout the state. Entry data is collected for each child at the time of 
eligibility determination and updated on a regular basis. Part B-eligible children are 
followed by NC's Department of Public Instruction. These data are also collected at the time 
of eligibility determination, entered and maintained at the local level using the Student 
Information Management System (SIMS). 



F. Expected Outcome: Service coordination for young children with special needs and their 
families will improve as a result of Smart Start. 

Measurement Strategy/Tool: focus groups 

Data Sources: parents of children with disabilities and professionals representing child 
care, early intervention, public preschool programs, public health, and other community 
agencies and programs 

Variables/Domains: awareness of services, barriers and supports to inclusion, changes in 
services delivery models, effects of Smart Start on early intervention services, and future 
directions of early intervention 

Data Collection Times: fall of 1994 
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Numbers: We will conduct six focus groups, 3 with parents and 3 with professionals, with 
approximately 10-12 participants per session. Two focus groups (one parent and one 
professional) will be held in each of three regions, western, central, and eastern North 
Carolina. 

Selection Process: The parents and professionals will be recruited through various agencies 
and programs within Smart Start counties. Using criteria identified by our project staff, 
agencies and programs will nominate parents and professionals as potential participants. 
Letters and consent forms will be sent to this list of people inviting them to participate. 



G. Expected Outcome: Family participation in and satisfaction with early intervention and 
expanded early childhood services for all children will increase. 

Measurement Strategy/Tool: family surveys 

Data Source: families of children with disabilities who are currently being served by 
Developmental Day programs, Early Childhood Intervention programs, or Public Preschools 
across North Carolina 

Variables/Domains: parental satisfaction with early intervention and expanded early 
childhood programs for all children, parental attitudes toward inclusion, impact of having 
children with disabilities on family functioning, and ratings of children’s abilities 

Data Collection Times: fall of 1994 (Oct-Dec) and one year later 

Numbers: approximately 360 families, 180 from Smart Start and 180 from non-Smart Start 
counties 

Selection Process: We will contact all Developmental Day programs. Early Childhood 
Intervention programs, and Public Preschools across North Carolina, inviting them to 
participate by sending us a list of I.D. numbers for all children with disabilities who are 
currently enrolled in their programs. We will then randomly select a portion of children 
from these lists and mail survey packets to be forwarded to the families. We will also have 
the programs to provide us with information on these selected children. 



H. Expected Outcome: The quality of child care arrangements for children with disabilities 
birth to 5 enrolled in regular child care programs will improve. 

NOTE: This outcome will be assessed through all three approaches to measurement of the 

broader Goal #3 (see above). Enrollment of children with special needs in regular child care 
programs will be documented. This will allow us to assess the quality of programs serving 
children with special needs and compare it to that of programs that only serve typically 
developing children as well as to compare quality changes over time. 
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Goal 4: North Carolina counties value all of their children and families 
by providing options, resources, and encouraging collaboration 
to help children and families reach their full potential. 



A. Expected Outcome: Inter-agency coordination and collaboration between service agencies 
in Smart Start counties will improve over time. 

Measurement Strategy/Tool: Network Analysis Survey 

Data Source: service administrators and providers will complete the survey 

Domains: awareness of other community services, amount of interaction with other 
agencies, referrals to and from other agencies, coordination in service provision and 
policy making, formal relationships with other agencies, and satisfaction with other 
community agencies 

Data Collection Times: annually in the winter 

Numbers: approximately 240 agency administrators and 480 agency service providers 

Selection Process: A list of 20 agencies and service organizations that primarily serve 
children and their families will be developed. Some of the agencies will be similar across 
all Smart Start counties, and others will be unique. For each agency listed, the director 
and a random sample of service providers will be asked to complete the survey. 

Other Details: The Network Analysis Survey will be mailed to participating agency 
directors and service providers. UNC’s County Evaluation Coordinators will be 
responsible for collecting the completed surveys. 



B. Expected Outcome: Over time, families in Smart Start counties/region will report that more 
services are available in the community, the services are more affordable, and the services are of 
higher quality. Family strengths will also increase over time. 

Measurement Strategy/Tool: Family Interview 

Data Source: families in each Smart Start county/region 

Domains: service availability and use, satisfaction with services, supportiveness of 
agency service providers, barriers to receiving services, choices in child care, child care 
costs, involvement in the community, family strengths 

Data Collection Times: annually during the fall and winter 
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Numbers: approximately 1500 families 

Selection Process: Families wall be selected with 3 different sampling strategies. First, 
questionnaires will be sent home to parents of children in all child care classrooms that 
are observed. This sample will enable us to address families’ options of child care and 
the affordability of child care. Second, families in the child service coordination caseload 
will be randomly selected. Families in this sample are most likely to be receiving 
multiple community services. This subsample will enable us to address issues regarding 
service availability, satisfaction, and barriers to services. Third, each local partnership 
will have an opportunity to nominate a target group of families whom the partnership 
believes is benefiting directly from Smart Start. This target group may include families 
who are participating in a particular program or families from particular neighborhoods 
that have been targeted by the partnership. Within the target group, a random sample of 
families will be interviewed. This strategy ensures that the opinions and experiences of 
families benefiting directly from local Smart Start efforts will be represented in the 
evaluation. 

Other details: All selected families will be asked the same set of questions. In the child 
care sample, the questions will be in the form of a written survey to be completed by 
families. In the other two samples of families, the same set of questions will be asked 
through an interview format (either face-to-face or by phone). 



C. Expected Outcomes: The number of agencies that have parent representatives on their 
advisory boards will increase in Smart Start counties/regions. The number of parents reported by 
center directors as participating in the center/home will increase over time. 

Measurement Strategy/Tool: Agency Survey, and Child Care Center Director Interview 

Data Source: agency administrators and child care center directors 

Domains: parent representation on advisory boards; parent participation in child care 
centers (e.g., volunteering, donating materials) 

Data Collection Times: annually in the fall 

Numbers: approximately 240 agency administrators who receive the Network Analysis 
Survey; child care directors in each of the 243 centers participating in the observations 

Selection Process: Agency administrators and child care directors who participate in the 
Network Analysis and child care observations will be surveyed or interviewed. 
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Procedural Issues 



Informed consent . For most evaluation strategies described in this report, informed 
consent will be obtained from participants prior to their participation in this evaluation. Families, 
children, child care directors and providers, and agency administrators and staff will be asked to 
participate in this study and will always have the option of declining to participate. Consent will 
not be obtained in those instances in which individually identifying information is not linked to 
the data. For example, data from an existing database that does not contain any personal 
identifiers can be used without obtaining an individual’s consent. Additionally, teacher 
checklists completed on a sample of kindergartners will not require parental consent because 
names of individual children will not be written on the checklist. 

Data collectors . Much of the data needed for this evaluation must be collected at the local 
level— counts of children and families served by programs, lists of agencies providing services, 
family interviews, observations of child care centers and family day care homes. In counties 
within an hour or two of Chapel Hill, project research assistants and investigators will collect 
most of these data. In more distant counties, we have hired County Evaluation Coordinators to 
collect information. When needed, Chapel Hill-based researchers will work with them in their 
counties. In three counties, we are “sharing” a data collector with the local partnership, which 
means that the person works approximately 50% effort collecting information that the local 
partnership desires and 50% effort collecting information for the statewide evaluation. This 
arrangement has been successful so far. We plan to have County Evaluation Coordinators 
“trade” counties for some types of data collection where anonymity of the data collector would 
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be helpful, for example, visits to child care programs in cases where the local data collector is 
well known to the program. 

Limitations of the Plan 

This evaluation is designed to address progress toward the Smart Start goals, as 
represented in the Goals Statement from the NC Partnership for Children, in the 18 originally 
funded Smart Start counties. The evaluation team’s primary responsibility is to provide an 
overall evaluation of Smart Start, although some evaluation information will be available within 
counties/regions that is specific to that area. For example, without compromising confidentiality 
almost all data can be summarized at the county level and shared with the local partnerships. 
Information such as the quality of child care in centers that are participating in Smart Start 
compared to those that are not participating will be available to local partnerships. 

However, this evaluation plan cannot address all of the local-level evaluation needs. The 
evaluation plan will not provide information on every project funded by local partnerships. 

Local partnerships may want to develop their own evaluation/monitoring system to obtain data 
regarding the progress of specific projects and to help them make future decisions about how best 
to use their funds. The evaluation team has been accessible to counties to provide technical 
assistance regarding local evaluation needs. Each partnership has an evaluation team contact 
person who is available to provide technical assistance regarding the development of local 
evaluation strategies. Additionally, all data collection measures (i.e., Family Interview) used by 
the statewide evaluation team will be available for use by local partnerships, if they want to 
supplement data from our randomly drawn samples with the same or similar data from samples 
of their own choosing. 
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One goal delineated by the NC Partnership for Children-that North Carolina’s state 
government, NC Partnership, and local partnerships will work collaboratively— is not addressed 
directly by this evaluation plan. This is a goal for which the local partnerships, state government, 
and the NC Partnership for Children are jointly responsible, and we think they are the best 
assessors of progress towards this goal. Our role might be to host forums or focus groups for 
discussion of the collaborative process and the challenges that remain to be solved to enable a 
smoother, more collaborative implementation. To date, however, we have focused our attention 
and planning on the child, family, and program level of evaluation, leaving open discussion of 
our role vis a vis this goal. In light of the fact that the evaluation team at UNC receives its 
funding from the Department of Human Resources, we could not be considered “independent” 
evaluators of this aspect of Smart Start. 

Finally, overnight results from Smart Start efforts should not be expected by the funding 
agencies nor the public. While we can document that more children and families are receiving 
services such as child care, immunizations, and home visits than in the year before Smart Start, 
the positive effects of these services will not be seen immediately. Comparing year to year 
progress on indicators such as child care quality, immunization rates, family strengths, and 
school readiness requires a sustained commitment over several years. The problems that resulted 
in the need for Smart Start funding were not created overnight, but through an accumulation of 
years of underfunded services, inappropriate services, and/or isolation from services. Smart Start 
is a pioneering effort to turn around those conditions. The efforts being undertaken by the 
partnerships are based on previous research, knowledge of their region’s needs, a broad-based 
consensus, and common sense. They can put new and/or improved projects in place in a year or 
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two, but the cumulative problems of children and families will take years to address effectively. 
The evaluation plan will assess the quality and extent of their efforts, both initially and ongoing, 
and measure child, family, and program outcomes over the years. 
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